Multimodal Language Processing For

Interfaces for mobile information access need to allow users flexibility in their choice of modes and interaction style in accordance with their preferences, the task at hand, and their physical and social environment. This paper describes the approach to multimodal language processing in MATCH (Multimodal Access To City Help), a mobile multimodal speech-pen interface to restaurant and subway information for New York City. Finite-state methods for multimodal integration and understanding enable users to interact using pen, speech, or dynamic combinations of the two, and a speech-act based multimodal dialogue manager enables mixedinitiative multimodal dialogue. 1. LANGUAGE PROCESSING FOR MOBILE SYSTEMS Mobile information access devices (PDAs, tablet PCs, next generation phones) offer limited screen real estate and no keyboard or mouse, making complex graphical interfaces cumbersome. Multimodal interfaces can address this problem by enabling speech and pen input and output combining synthetic speech and graphics (See [1] for a detailed overview of previous work on multimodal input and output). Furthermore, since mobile devices are used in situations involving different physical and social environments, tasks, and users, they need to allow users to provide input in whichever mode or combination of modes are most appropriate given the situation and the user’s preferences. Our testbed multimodal application MATCH (Multimodal Access To City Help) allows all commands to be expressed either by speech, by pen, or multimodally. This is achieved by capturing the parsing, integration, and understanding of speech and gesture inputs in a single multimodal grammar which is compiled into a multimodal finite-state device. This device is tightly integrated with a speech-act based multimodal dialog manager enabling users to complete commands either in a single turn or over the course of a number of dialogue turns. In Section 2 we describe the MATCH application. In Section 3, we describe the multimodal language processing architecture underlying MATCH. 2. THE MATCH APPLICATION Urban environments present a complex and constantly changing body of information regarding restaurants, cinema and theatre schedules, transportation topology, and timetables. This information is most valuable if it can be delivered effectively while mobile, since users needs change while they are out and the information itself is dynamic (e.g. train times change and shows get cancelled). Thanks to AT&T labs and DARPA ITO (contract No. MDA972-99-30003) for financial support. MA enables for New pen com less net ing rest street in mands play wi of the t rants u